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SUMMARY 


^/Research  on  expert-novice  differences  has  mainly  focused  on  how 
experts  solve  familiar  problems.  We  know  far  less  about  the  skills  and 
knowledge  used  by  experts  when  they  are  confronted  with  novel  problems 
within  their  area  of  expertise.  This  report  discusses  a  study  in  which 
verbal  protocols  were  taken  from  subjects  of  various  expertise  design¬ 
ing  an  experiment  in  an  area  they  were  unfamiliar  with.  The  results 
showed  that  even  when  domain  knowledge  is  lacking,  experts  solve  a 
problem  within  their  area  of  expertise  by  dividing  the  problem  into  a 
number  of  subproblems  that  are  solved  in  a  specified  order.  The  lack 
of  domain  knowledge  is  compensated  for  by  using  abstract  knowledge 
structures  and  domain-specific  strategies.  The  results  suggest  that 
experts  are  confronted  with  novel  problems,  they  can  bring  to  bear 


various  types  of  knowledge  and  strategies  that  enable  them  to  outper¬ 
form  novices.  ^  ^  £-  ‘  >  '■  '  *  ’'•  U^vs.x*.  ( r.  c_.  ,  - 
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Rap. nr.  IZF  1990  B-14  Instituut  voor  Zlnculgfyalologle  TNO, 

Soesterberg 


Ho*  expert*  een  nieuv  probleem  blnnen  hun  expert lsegebled  oploeaen 
J.M.C.  Schraagen 

SAMENVATTING 

Ondarzoek  op  het  gabled  van  verschlllen  tussen  beginners  an  experts 
heeft  zlch  tot  nu  toe  vooral  gerlcht  op  hoe  experts  bekende  problemen 
oplossen.  Veel  minder  Is  bekend  over  de  kennla  en  vaardlgheden  die 
experts  gebrulken  wanneer  ze  geconfronteerd  worden  met  nleuwe  proble- 
men  blnnen  hun  vakgebled.  In  dlt  rapport  wordt  verslag  gedaan  van  een 
studie  waarin  proefpersonen  van  verschlllende  expertlsenlveaus  hardop 
denkend  een  onderzoek  moesten  opzetten  op  een  voor  hen  onbekend 
gebied.  De  resultaten  laten  zlen  dat  zelfs  wanneer  domeinkennls 
ontbreekt,  experts  een  probleem  blnnen  hun  vakgebled  oplossen  door  dat 
probleem  In  een  aantal  subproblemen  op  te  delen  en  die  vervolgens  In 
een  vaste  volgorde  op  te  lossen.  Het  gebrek  aan  domeinkennls  wordt 
gecompenseerd  door  abstracts  kennlastructuren  en  domeinspeclfieke 
strategies  te  gebrulken.  De  resultaten  suggereren  dat  wanneer  experts 
met  nleuwe  problemen  worden  geconfronteerd,  ze  verschlllende  soorten 
kennls  en  strategies  kunnen  gebrulken  die  hen  in  staat  stellen  beter 
te  presteren  dan  beginners. 
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1  INTRODUCTION 

In  the  past  ten  years,  research  on  problem  solving  has  mainly  focused 
on  differences  In  the  way  experts  and  novices  structure  their  knowl¬ 
edge  (for  reviews,  see  Glaser,  1984;  Greeno  &  Simon,  1988;  Van  Lehn, 
1989).  This  research  has  clearly  shown  that  the  expert's  knowledge 
base  Is  more  abstract,  more  principled,  and  more  organized  for  use 
than  the  novice's  knowledge  base. 

However,  several  Important  questions  have  been  neglected  In  the 
research  mentioned  above,  In  a  recent  review  on  problem  solving  and 
reasoning,  Greeno  and  Simon  (1988)  mentioned  as  one  of  tho  unanswered 
questions  the  Interactive  development  and  utilization  of  general  and 
specific  skills.  For  Instance,  when  confronted  with  novel  problems 
within  their  domain  of  expertise,  do  experts  resort  to  general  strat¬ 
egies  (or  "weak  methods")  and  behave  like  novices,  or  do  they  transfer 
more  task- specific  strategies  to  these  novel  problems  and  perform 
better  than  novices?  A  few  studies  have  been  carried  out  on  the 
problem  solving  skills  experts  use  when  confronted  with  novel  problems 
(e.g.,  Adelson  &  Soloway,  1985;  Voss,  Gresne,  Post  &  Penner,  1983; 
Larkin,  1983).  The  results  of  these  studies  suggest  that  experts  have 
learned  moderately  general  strategies  such  as  mental  simulation  that 
are  nevertheless  specific  to  particular  domains,  for  Instance  software 
design  (Adelson  &  Soloway,  1985).  When  they  are  confronted  with  novel 
problems,  experts  use  these  strategies  to  solve  these  novel  problems. 
Since  novices  do  not  use  these  strategies,  they  have  to  search  more 
and  hence  perform  poorer  than  experts. 

Besides  using  task-specific  strategies,  a  second  way  in  which  experts 
could  perform  better  than  novices,  when  confronted  with  novel  prob¬ 
lems,  is  by  using  their  more  abstract  and  more  principled  knowledge 
base.  Novel  problems  could  remind  experts  of  previously  solved  prob¬ 
lems  that  are  similar  to  the  current  problem  In  an  abstract  way.  The 
study  by  Voss  et  al.  (1983)  showed  that  experts  whose  domain  knowledge 
was  lacking,  still  came  up  with  more  general  subproblems  then  novices. 
Evidence  for  the  importance  of  how  knowledge  is  represented  also  comes 
from  studies  of  analogical  transfer  (Cick  &  Holyoak,  1980,  1983; 
Holyoak  &  Koh,  1987;  Novlck,  1988).  In  this  research  area,  a  dis¬ 
tinction  Is  made  between  structural  and  surface  problem  features. 
Structural  features  are  abstract,  whereas  surface  features  are  more 
literal.  Novlck  (1988)  has  shown  that  since  the  representations  of 
experts  include  both  surface  and  structural  features,  spontaneous 
positive  transfer  occurs  in  experts'  problem  solving  attempts  when  the 
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target  problem  and  lta  analogue  share  structural  features  but  are 
superficially  dissimilar.  Since  the  representations  of  novices  Include 
only  surface  features  (e.g.,  Chi,  Feltovich  &  Glaser,  1981;  Adelson, 
1981),  positive  transfer  does  not  occur  in  novices’  problem  solving 
when  the  target  problem  and  its  analogue  share  structural  features  but 
are  superficially  dissimilar.  Since  research  In  this  area  does  not 
typically  make  use  of  verbal  protocols,  It  remains  unclear  what 
strategies  experts  use  to  determine  the  appropriate  structural  fea- 
tures  in  a  problem,  and  what  strategies  they  use  to  adapt  the  analogue 
to  the  target  problem. 

In  conclusion,  although  there  has  been  some  research  on  how  experts 
transfer  their  knowledge  to  novel  problem  situations,  the  Interaction 
between  representations  and  strategies  Is  often  left  unclear.  Mostly, 
the  focus  has  been  on  either  strategies  or  representations,  but  their 
joint  contribution  has  not  been  studied  in  complex,  real-world  prob¬ 
lems.  The  present  study  Is  an  attempt  to  remedy  this  situation. 

The  question  how  experts  solve  novel  problems  may  be  viewed  as  a 
question  about  the  transfer  between  pre- experimental  knowledge  and 
performance  on  a  particular  task.  The  question  of  the  transfer  of 
expert  knowledge  to  novel  problems  is  an  important  one,  both  for 
theoretical  and  practical  reasons.  Theoretically,  questions  dealing 
with  the  transfer  of  knowledge  and  skills  have  important  implications 
for  theories  of  knowledge  representation  (Singley  &  Anderson,  1989). 
Practically,  finding  evidence  for  positive  transfer  of  expert  knowl¬ 
edge  to  novel  situations  could  have  educational  implications .  The 
strategies  and  representations  used  by  experts  could  be  made  explicit 
and  perhaps  successfully  taught  to  novices.  A  second  practical  appli¬ 
cation  could  be  the  incorporation  of  these  strategies  and  representa¬ 
tions  in  expert  systems,  thereby  making  them  less  brittle  than  they 
are  now. 

The  rest  of  this  paper  is  structured  as  follows.  The  next  section  will 
outline  a  theoretical  framework  that  will  allow  us  to  make  general 
predictions  and  provide  the  vocabulary  with  which  to  describe  the  task 
to  be  studied  here,  namely  designing  an  experiment  in  the  area  of 
sensory  psychology.  This  task  is  described  in  detail  in  the  section 
following  the  theoretical  framework.  After  this  task  analysis,  we  are 
able  to  derive  a  model  of  expert  problem  solving  in  this  particular 
task  domain.  The  model  is  operationalized  in  terms  of  a  coding  scheme 
for  the  verbal  protocols  used  for  tasting  the  model.  In  the  results 
section,  the  model  is  tested.  Finally,  the  general  discussion  will 


9 


consider  che  implication*  of  Che  resulCe  for  Che  cheoreclcal  frame¬ 
work,  as  well  as  for  some  pracClcal  Issues. 


1.1  ThaoreClcal  framework 

The  cheoreclcal  framework  outlined  below  contains  a  number  of  elements 
Chat  are  derived  from  a  variety  oi  sources  (Anderson,  1983,  1987; 
Laird,  Newell  &  Rosenbloom,  1987;  Jansweijer,  1988;  Hamel,  1990). 

Current  theories  of  cognition  (e.g.,  Anderson,  1983)  attach  great 
importance  to  hierarchical  goal  structures .  Goal  structures  specify 
what  goals  have  to  be  accomplished  in  order  to  carry  out  a  task.  They 
function  as  an  efficient  sequence  of  steps  for  carrying  out  a  task. 
For  instance,  when  solving  physics  problems,  it  is  efficient  to 
convert  all  initial  data  into  SI -units  (degrees  Fahrenheit  into 
degrees  Kelvin,  etc.).  Beginners  frequently  forget  this  step,  or  carry 
it  out  in  the  middle  of  solving  equations,  whereas  experts  have 
learned  to  accomplish  this  goal  before  solving  the  equations 
(Jansweijer,  1988).  Hence,  goal  structures  control  behavior  and 
provide  task  decomposition  knowledge.  That  is,  they  either  divide  a 
task  into  a  number  of  subtasks  or  they  directly  solve  a  subtask  by 
applying  domain  knowledge.  When  all  the  goals  are  accomplished  in  the 
specified  order,  problem  solving  follows  a  structured  path.  Goal 
structures  are  knowledge  structures  that  are  initially  derived  from 
task  instructions  and  experience  with  similar  problems.  With  practice 
on  a  particular  task,  goal  structures  grow  more  elaborate  and  more 
structured. 

The  goal  structure  itself  is  stored  in  long-term  memory  (LTM)  and  is 
retrieved  after  the  task  specifications  are  understood.  The  goal 
structure  is  deposited  in  a  limited -capacity  working  memory  (WM) .  Only 
one  goal  is  currently  in  the  focus  of  attention,  but  closely  linked 
goals  will  probably  also  receive  some  activation  (Anderson,  1983) .  One 
of  the  consequences  of  this  limited  capacity  is  that  subgoals  cannot 
be  pursued  indefinitely  when  knowledge  is  lacking,  or  else  the  orig¬ 
inal  goal  will  be  forgotten. 

The  concept  of  a  goal  structure  cannot  by  Itself  explain  why  experts 
may  have  developed  dome  In -dependent  strategies  (or  heuristics)  that 
they  can  use  in  novel  problem  situations.  Fixed  goal  structures 
control  behavior  In  routine  problem  solving.  However,  when  knowledge 
is  insufficient  and  an  Impasse  Is  encountered  (Laird,  Newell  & 
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Rosenbloom,  1987),  a  particular  goal  cannot  ba  accomplished  any  more. 
In  these  situations,  experts  may  have  developed  heuristics  that  tell 
them  what  to  do  next.  These  heuristics  dynamically  update  the  goal 
structure  during  problem  solving,  for  Instance  by  setting  subgoals  to 
repair  the  impasse.  Problem  solving  Is  temporarily  halted  as  the 
requisite  domain  knowledge  is  assembled  In  another  problem  space. 

The  final  element  In  the  theoretical  framework  is  a  structure  that 
contains  all  the  results  of  problem  solving  carried  out  so  far.  This 
structure  Is  called  the  "problem  conception”  (Hamel,  1990),  or  problem 
representation.  1  will  assuma  the  problem  conception  to  be  schemati¬ 
cally  organized  (cf.  Van  Lahn,  1989).  This  schema  Is  a  knowledge 
structure  that  is  selected  when  the  problem  description  Is  read, 
stored  in  working  memory,  and  gradually  elaborated  with  domain  knowl¬ 
edge  during  problem  solving.  The  problem  conception  schema  is  domain- 
specific  yet  general  at  the  same  time.  It  is  domain-specific  because 
it  specifies  what  domain  knowledge  should  be  included  in  the  open 
slots  of  the  schema.  However,  it  is  also  general  in  that  the  nature  of 
the  knowledge  and  the  relations  among  slots  are  specified  in  advance, 
independent  of  the  particular  problem  to  be  solved.  By  adapting  the 
schema  to  a  problem,  missing  data  are  supplied  by  a  process  of  elabor¬ 
ation,  individual  data  are  identified  as  values  of  variables,  and 
irrelevant  details  are  ignored.  When  the  schema  is  successfully 
adapted,  the  problem  is  said  to  be  understood.  The  problem  can  now  be 
solved  using  procedural  knowledge  contained  in  the  schema. 

During  actual  problem  solving,  the  goal  structure  and  the  problem 
conception  are  intimately  connected.  The  goal  structure  controls  the 
selection  and  refinement  of  the  problem  conception  schema  by  applying 
domain  knowledge  to  a  task.  In  turn,  the  domain  knowledge  contained  in 
the  problem  conception  schema  allows  a  particular  goal  to  be  accom¬ 
plished.  Therefore,  an  impoverished  problem  conception  schema  will 
lead  to  less  structursd  problem  solving,  involving  %  frequent  appeal 
to  subgoals . 

The  theoretical  framework  described  above  allows  us  to  make  predic¬ 
tions  about  what  happens  when  experts  are  confronted  with  novel 
problems . 

First,  ths  literature  on  expert-novice  differences  has  clearly  shown 
that  experts  have  more  elaborate  goal  structures  than  novices 
(Jsnswsijsr,  1988).  Hence,  their  problem  solving  will  be  mors  struc¬ 
tured  than  that  of  novices.  On  the  other  hand,  experts  whose  domain 
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knowledge  Is  lacking  becausa  they  are  confronted  with  novel  problems, 
will  show  less  structured  problem  solving  than  experts  whose  domain 
knowledge  is  not  lacking. 

Second,  novices  have  not  had  enough  experience  with  a  particular  task 
to  have  developed  heuristics  as  powerful  as  the  experts'.  Hence, 
experts  will  make  use  of  domain- dependent  strategic  knowledge  when 
confronted  with  novel  problems,  whereas  novices  have  to  rely  on 
domain- independent  strategic  knowledge  (or  "weak  methods"),  the 
domain-dependent  strategic  knowledge  will  constrain  search  to  a 
greater  extent  than  the  weak  methods.  This  greater  search  constraint 
will  prevent  the  experts  from  falling  back  on  seemingly  random, 
novice -like  problem  solving  behavior. 

Third,  experts  have  a  better  integrated  and  more  abstract  problem 
conception  schema  than  novices.  The  use  of  a  well- integrated  problem 
conception  schema  implies  that  experts  will  not  suffer  from  working 
memory  overload  as  often  as  novices.  When  solving  novel  problems, 
experts  will  frequently  go  over  the  same  goal  again  and  again,  because 
relevant  domain  knowledge  is  lacking  and  has  to  be  assembled  in 
another  problem  space.  Hence,  their  problem  conception  schema  will  be 
successively  refined,  showing  a  pattern  of  progressive  deepening  (Da 
Groot,  1978;  Kant  &  Newell,  1984).  More  important,  when  confronted 
with  novel  problems,  experts  will  be  able  to  use  the  general  elements 
in  their  problem  conception  schema  when  they  adapt  the  schema  to  the 
problem  and  hence  come  up  with  structurally  instead  of  superficially 
relevant  solutions. 

The  following  section  will  use  the  concepts  defined  above  in  describ¬ 
ing  the  task  subjects  had  to  carry  out  in  the  present  study. 


1.2  Designing  an  experiment  in  the  area  of  sensory  psychology 

The  problem  solving  domain  investigated  in  this  study  is  that  of 
designing  an  experiment  in  the  area  of  sensory  psychology.  The  follow¬ 
ing  paragraphs  will  describe  the  task  of  designing  experiments  by  both 
using  empirical  sources,  theoretical  analyses,  and  handbooks . 

Designing  experiments  is  an  instance  of  the  generic  task  of  design. 
This  classification  is  based  on  properties  of  the  input,  the  expected 
output,  and  the  nature  of  the  operation  taking  place  to  map  input  to 
output  (Staala,  1990).  The  input  to  experimental  design  is  a  research 
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question  containing  specifications.  The  output  Is  an  object,  the 
research  plan,  that  conforms  to  these  specifications.  Generic  tasks 
share  the  same  goal  structures  and  the  same  types  of  domain  knowledge 
(Chandrasekaran,  1983).  Independent  research  in  various  domains  has 
found  that  design  tasks  are  often  decomposed  into  the  following 
subtasks  (Brown  &  Chandrasekaran,  1986;  Malhotra,  Thomas,  Carroll  & 
Miller,  1980;  Marcus,  Stout  &  McDermott,  1987;  Mittal,  Dym  &  Morjaria, 
1986): 

1  test  specifications  for  incompleteness  or  inconsistency 

2  generate  or  extend  a  partial  solution 

3  test  the  adequacy  of  the  solution  by  matching  it  with  constraints 

4  refine  the  solution  by  resolving  violated  constraints 

By  means  of  these  subtasks,  the  input  is  .pped  to  the  output.  Some  of 
the  pragmatic  problems  (Steels,  1990)  associated  with  design  tasks  are 
the  incompleteness  of  the  specifications,  the  large  number  of  partial 
solutions  possible,  and  the  limited  memory  available  for  storing 
structure.  These  pragmatic  problems  determine  to  a  Large  extent  the 
strategies  and  types  of  domain  knowledge  used  by  problem  solvers.  For 
instance,  the  incompleteness  of  the  specifications  forces  the  problem 
solver  to  test  the  specifications  by  validating  the  data,  broadening 
or  restricting  the  context,  classifying  the  data,  or  deducing  addi¬ 
tional  features  based  on  class  membership.  The  large  number  of  partial 
solutions  possible  implies  a  structuring  of  solutions  in  terms  of 
typical  features  and  not  in  terms  of  necessary  and  sufficient  condi¬ 
tions.  The  limited  memory  available  forces  the  problem  solver  to 
progressively  deepen  the  solution.  This  general  analysis  of  design 
tasks  will  next  be  applied  to  design  of  experiments. 

In  handbooks  on  experimental  design  (e.g.,  Kerllnger,  1973,  p.300), 
one  often  finds  the  following  two  general  goals  that  together  consti¬ 
tute  the  task  of  designing  an  '-xpsriment: 

1  Answer  the  research  question 

2  Control  all  sources  of  variance 

Based  on  the  task  decomposition  of  the  generic  task  of  design,  I  will 
assume  that  the  goal  of  answering  the  research  question  is  accom¬ 
plished  by  understanding  the  problem,  selecting  a  paradigm,  and 
pursuing  that  paradigm.  Understanding  the  problem  is  the  equivalent  of 
testing  the  specifications,  selecting  a  paradigm  is  equivalent  with 
generating  a  partial  solution,  and  pursuing  a  paradigm  is  equivalent 
with  refining  the  solution.  The  goal  of  controlling  all  sources  of 
variance  Is  equivalent  with  testing  the  adequacy  of  the  solution. 
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The  notion  of  a  paradigm  aa  the  knowledge  structure  that  guldea 
experts'  problem  solving  when  designing  experiments,  was  derived  by 
analogy  with  the  medical  domain.  From  previous  work  In  the  medical 
domain  (e.g.,  Feltovlch  &  Barrows,  1984),  it  was  clear  that  medical 
experts  used  complex  knowledge  structures  when  processing  and  recall¬ 
ing  medical  information.  Feltovlch  and  Barrows  referred  to  these 
knowledge  structures  as  "illness  scripts”.  In  the  authors'  words,  a 
clinician  "attempts  to  represent  and  understand  a  patient  problem  by 
constructing  an  Integrated  script  or  scenario  for  how  the  patient's 
condition  ••  ame  to  be,  its  major  points  of  malfunction,  and  its  subse¬ 
quent  assoc<ated  consequences"  (p.139).  In  terms  of  our  theoretical 
frameworx,  the  illness  script  may  be  viewed  as  an  example  of  a  problem 
conception  schema. 

When  designing  experiments,  it  is  often  useful  to  classify  a  particu¬ 
lar  research  question  as  an  instance  of  a  more  general  question  that 
may  be  solved  by  some  general  research  plan  (Friedland,  1979;  Johnson, 
Nachtsheim  &  Zualkerman,  1987)  or  paradigm.  For  instance,  a  research 
question  on  "how  well  people  are  able  to  remember  faces  of  criminals 
they  have  only  seen  for  a  short  moment"  may  be  classified  as  an 
instance  of  the  more  general  question:  "how  well  are  people  able  to 
recognize  stimuli".  This  general  question  then  evokes  a  "recognition 
paradigm"  from  memory  that  specifies  what  steps  have  to  be  taken  to 
answer  this  question  in  a  scientific  way.  More  specifically,  a  para¬ 
digm  is  a  general  research  plan  containing  a  specification  of  the 
subjects  and  the  independent  and  dependent  variables  to  be  used  in  the 
experiment.  A  paradigm  may  also  contain  specifications  of  the  instruc¬ 
tions  to  subjects,  the  setting  where  the  experiment  is  carried  out, 
the  outcome  of  the  experiment,  and  control  variables  (to  be  discussed 
in  th»<  next  paragraph).  Usually  a  subject  is  first  selected,  then 
receives  a  treatment  in  the  form  of  an  Independent  variable,  and 
finally  a  particular  aspect  (the  dependent  variable)  is  measured. 
Hence,  there  is  a  temporal  ordering  in  the  elements  constituting  the 
paradigm.  Since  paradigms  are  applicable  in  a  wide  range  of  situ¬ 
ations,  they  are  indexed  with  respect  to  fairly  general  goals.  For 
Instance,  a  recognition  paradigm  accomplishes  the  goal  of  finding  out 
whether  someone,  when  presented  with  one  or  more  alternatives,  is 
familiar  with  those  alternatives.  A  multidimensional  scaling  paradigm 
may  be  Indexed  under:  "this  paradigm  accomplishes  the  goal  of  describ¬ 
ing  a  large  number  of,  often  perceptual,  stimuli  into  a  fewer  number 
of  underlying  dimensions*.  Knowledge  about  paradigms  may  be  considered 
a  catalog  of  hierarchically  organized  prototypes,  or  "skeletal  plans" 
(Friedland  &  Iwasaki,  1985).  The  hierarchical  way  of  structuring 
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enables  the  problem  solver  Co  reduce  the  number  of  solutions  to  search 
for,  which  is  particularly  useful  in  design  tasks,  as  discussed  above. 

The  goal  of  controlling  all  sources  of  variance  is  accomplished  by 
generating  design  principles  that  minimize  the  error  variance  and 
maximize  the  systematic  variance  in  an  experiment.  These  general  goals 
are  accomplished  in  turn  by  more  specific  goals  such  as  experimental 
control,  reliable  measurement,  using  homogeneous  groups  of  subjects, 
increasing  sample  size,  and  using  widely  different  experimental 
conditions.  The  goal  of  experimental  control  is  still  fairly  general 
and  Is  achieved  by  more  specific  goals  such  as  "avoid  carryover 
effects”.  This  particular  goal  may  be  accomplished  by  counterbalancing 
conditions.  Control  of  variance  is  a  goal  familiar  to  all  students  of 
experimental  psychology,  and  ways  of  achieving  this  goal  may  be  found 
in  any  textbook  on  this  subject  (e.g.,  Neale  &  Liebert,  1980).  The 
general  design  principles  may  be  viewed  as  constraints  against  which 
the  partial  solution  is  tested. 

One  of  the  alms  of  our  protocol  analyses  was  to  identify  the  different 
strategies  used  by  subjects  whenever  they  encountered  Impasses  due  to 
a  lack  of  knowledge.  In  principle,  knowledge  may  be  lacking  for  each 
of  the  goals  mentioned  above.  However,  I  was  not  interested  in  prob¬ 
lems  beginners  might  have  in  understanding  the  problem  statement, 
since  in  that  case  they  would  not  even  be  able  to  start  designing  an 
experiment.  I  therefore  chose  a  problem  that  all  subjects  would  in 
principle  be  able  to  understand,  viz.  a  problem  that  required  knowl¬ 
edge  of  soft  drinks  and  their  taste.  This  choice  of  problem  allowed  us 
to  focus  on  the  knowledge  and  strategies  subjects  would  bring  to  bear 
when  actually  designing  an  experiment. 

The  primary  Interest  in  this  study  was  in  how  experts  solve  novel 
problems  within  thsir  domain  of  expertise.  The  domain  of  expertise  in 
this  case  was  designing  psychological  experiments.  In  order  to  ident¬ 
ify  what  is  specific  for  this  particular  group  of  experts,  the  study 
Included  subjects  with  less  experience  with  designing  experiments 
(i.e.,  beginners  and  Intermediates)  and  subjects  with  more  domain- 
specific  knowledge  (i.e.,  domain  experts).  Hence,  the  other  three 
groups  served  as  controls.  For  the  domain  experts,  the  problem  they 
had  to  solve  was  relatively  easy,  although  not  trivial.  The  use  of 
more  than  two  groups  of  subjects  of  varying  expertise  was  inspired  by 
the  study  of  Voss  at  al,  (1983).  It  avoids  a  problem  usually  associ¬ 
ated  with  expert-novice  studies,  namely  that  experts  may  be  very 
different  from  novices  in  other  respects  than  their  greater  experl- 
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one*,  for  Instance  In  intelligence  or  motivation.  By  uaing  more 
groups,  the  transition  from  novice  to  expert  could  be  viewed  in  a  more 
gradual  way,  and  allowed  us  to  make  more  comparisons  among  groups, 
thereby  helping  to  "unconfound"  some  of  the  expert-novice  differences. 


2  METHOD 

2.1  Overview  of  the  methodology  used  in  the  present  atuti^ 

The  knowledge  and  strategies  used  by  subjects  were  assessed  by  col¬ 
lecting  verbal  protocols  of  subjects  while  designing  an  experiment. 
The  analysis  of  verbal  protocols  requires  a  coding  scheme  by  means  of 
which  statements  can  be  classified  into  particular  categories.  In 
developing  a  coding  scheme,  the  researcher  should  follow  particular 
rules  (see  Ericsson  &  Simon,  1984).  For  instance,  a  coding  scheme 
should  not  be  based  on  the  protocols  the  researcher  is  Interested  in, 
but  rather  on  a  task  analysis.  Furthermore,  the  statements  used  for 
developing  the  coding  scheme  should  be  scored  Independently  of  each 
other.  This  study  adopted  the  following  procedure: 

1)  Protocols  of  subjects  solving  a  similar  problem  as  in  this  study 
were  segmented  into  units  corresponding  to  sentences,  or,  in  some 
cases,  larger  idea  units.  Each  unit  was  typed  on  a  card.  The 
resultant  deck  of  58  cards  was  given  to  six  other  subjects  who  had 
not  solved  the  problem  but  who  were  familiar  with  the  area  of 
experimental  design.  Cards  were  presented  to  the  subjects  in  a 
random  order,  thus  ensuring  independent  scoring  of  each  unit.  These 
subjects  were  asked  to  sort  the  cards  into  as  many  categories  as 
they  thought  appropriate. 

2)  Categories  were  reduced  by  cluster  analysis.  To  this  end,  similar¬ 
ity  matrices  were  developed  based  on  the  categories  subjects  came 
up  with.  Two  units  received  a  similarity  score  of  1  when  they  were 
placed  in  the  same  category  and  a  score  of  0  when  they  were  placed 
in  different  categories.  These  similarity  matrices  were  averaged 
for  all  six  subjects  and  analyzed  by  means  of  a  hierarchical 
cluster  analysis.  The  results  of  the  cluster  analysis  showed  four 
categories  that  were  named  as  follows: 
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a)  understand  problem 

b)  operationalize  variables  (subjects,  ( independent  variables)  c) 
plan  (sequence  of  events) 

d)  validity  issues  (e.g. ,  carry-over  affects) 

Further  analysis  showed  that  these  categories  could  fairly  objecti¬ 
vely  be  established  by  looking  for  particular  key  words  (e.g., 
words  such  as  'identify',  recognize',  and  'taste'  indicated  problem 
understanding;  sequences  of  'then  ...  and  then*  indicated  the  plan 
for  data  collection;  words  such  as  'randomize'  and  'counterbalance' 
clearly  indicated  validity  issues). 

Hence,  the  categories  themselves  and  the  attribution  of  statements 
to  these  categories  were  established  by  fairly  objective  proce¬ 
dures,  thus  ensuring  sufficient  reliability  of  coding. 

3)  Based  on  a  task  analysis  (see  above),  these  four  categories  were 
slightly  modified  and  abstracted.  This  modification  resulted  in  the 
following  four  goals  that  are  sequentially  accomplished  in  the  task 
of  designing  experiments: 

a)  understand  problem 

b)  select  paradigm 

c )  pursue  paradigm 

d)  control  variance 

This  goal  structure  represents  an  "expert  model"  of  problem  solving 
in  the  area  of  designing  experiments. 

4)  Finally,  in  order  to  be  able  to  classify  actual  protocol  state¬ 
ments,  a  coding  scheme  was  developed.  The  goal  structure  mentioned 
above  was  extended  with  the  following  categories: 

a)  evaluation  statements,  whenever  there  is  insufficient  knowledge 
to  choose  among  two  or  more  knowledge  structures; 

b)  task-oriented  statements,  dealing  with  task  requirements,  ques¬ 
tions  to  the  experimenter,  and  the  evaluation  of  the  task  as  a 
whole ; 

c)  monitoring  statements  or  meta-comments,  when  subjects  report 
about  their  own  problem-solving  processes.  These  verbalizations 
are  often  of  limited  value,  since  they  do  not  direct  subsequent 
problem  solving  behavior  (Ericsson  &  Simon,  1984) . 

The  resulting  goal  structure  for  the  task  of  designing  experiments  is 
shown  in  Fig.  1. 
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menta  (arrows  Indicate  order  In  which  goals  are  accom¬ 
plished)  . 


The  coding  scheme  specifies  how  the  goal  structure  Is  manifested  In 
the  verbal  protocols.  Note  that  the  categories  in  the  coding  scheme 
were  developed  on  the  basis  of  a  pilot  study  and  not  on  the  basis  of 
the  protocols  to  be  discussed  In  this  study.  The  full  coding  scheme, 
with  examples  from  each  category,  is  included  In  Appendix  A.  By  using 
the  examples  and  the  key  words  underlined,  the  experimenter  was  able 
to  assign  statements  to  categories  in  a  fairly  objective  way.  Hence, 
no  second  coder  was  used  to  assess  inter-rater  reliability. 

Although,  according  to  the  task  analysis,  the  goals  are  sequentially 
accomplished,  backing  up  to  an  immediately  preceding  goal  is  allowed, 
because  "activation  spreading  from  tha  current  goal  will  maintain  in 
working  memory  the  most  closely  linked  goals"  (Anderson,  1983,  p.161). 
We  may  therefore  expect  to  see  these  associative  switches  between 
neighboring  goals  in  verbal  protocols. 
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2.2  Materials 

All  subjects  received  the  following  problem: 

The  manufacturer  of  Coca  Cola  wanta  to  Improve  hia 
product.  Recently,  he  hea  received  complaints  that  Coca 
Cola  doea  not  taate  aa  good  any  more  aa  it  used  to. 
Therefore,  he  wants  to  investigate  what  It  la  exactly 
that  people  taste  when  they  drink  Coca  Cola.  In  order  to 
be  able  to  make  a  comparison  with  the  competitors,  Pepsi 
Cola  and  a  house  brand  are  Included  In  the  study  aa 
well.  The  manufacturer  has  indicated  that  'taste'  may  be 
defined  very  broadly  in  this  study.  The  study  will  be 
conducted  by  a  bureau  for  market  research.  The  manufac¬ 
turer  thinks  of  the  entire  Dutch  population  as  the 
target  population. 

Please  indicate  as  detailed  as  possible  how,  according 
to  you,  such  a  study  would  look  like.  You  may  be  able  to 
come  up  with  more  than  one  solution.  In  that  case,  do 
not  hesitate  and  name  all  of  them! 

The  problem  description  was  deliberately  kept  vague,  in  order  to  bring 
out  differences  between  subjects  in  the  way  they  structured  the  prob¬ 
lem,  using  their  knowledge  of  paradigms.  In  particular,  the  problem 
was  vague  on  the  cause  of  the  complaints  the  cola  manufacturer  re¬ 
ceived  and  on  whether  the  type  of  study  he  proposes  logically  follows 
from  the  complaints  he  has  received.  The  problem  description  also 
contained  a  number  of  details  that  subjects  mey  change  or  abstract 
from.  These  details  concern  the  other  cola  brands,  the  broad  defini¬ 
tion  of  taste,  the  bureau  for  market  research,  and  the  target  popula¬ 
tion.  In  reality,  researchers  are  often  confronted  with  questions  that 
are  ambiguous,  unclear,  implicit  as  far  as  the  main  problem  is  con¬ 
cerned,  and  loaded  with  details. 

Subjects  received  the  following  think  aloud  instructions  on  paper 
(based  on  Ericsson  &  Simon,  1984): 

Try  to  think  eloud  while  performing  the  task.  By  this  I 
mean  that  you  tell  everything  from  the  moment  the  task 
begins  until  the  end  of  the  task.  I  will  ask  you  to 
constantly  talk  aloud  during  this  period.  I  do  not  want 
you  to  plan  ahead  what  you  are  going  to  say.  Act  as  if 
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you  calk  Co  yourself.  It  la  of  tha  utmoet  lmporCanca 
Chat  you  contlnua  Calking.  Whan  you  fall  allont  for  an 
axcandad  period  of  time,  tha  experimenter  will  aak  you 
to  a Cart  talking  again. 

Subjects  did  not  have  any  trouble  thinking  aloud  while  aolvlng  the 
problem . 


2.3  Subjects 

Four  categories  of  aubjecta  were  distinguished: 

1)  Beginners  (Beg):  undergraduatea  majoring  in  either  experimental 
psychology  (N-5)  or  In  methodology  (N-4) ;  the  beginners'  experience 
with  designing  experiments  was  limited  to  one  or  two  experiments. 

2)  Intermediates  (Int):  graduate  atudenta  In  experimental  psychology 
(N-2)  or  In  methodology  (N-l);  the  Intermediates'  experience  with 
designing  experiments  was  limited  to  three  or  four  experiments. 

3)  Design  experts  (DesExp):  subjects  with  at  least  ten  years  of  ex¬ 
perience  In  designing  experiments  In  various  areas,  except  in  tha 
area  of  sensory  psychology  (N-3) . 

4)  Domain  experts  (DomExp) :  subjects  with  at  least  ten  years  of  ex¬ 
perience  In  designing  experiments  In  the  area  of  sensory  psychology 
(N-4) . 


2.4  Procedure 

Subjects  were  tested  Individually  in  a  quiet  room  at  their  own  or  the 
experimenter's  office.  The  experimenter  told  them  that  he  was  Inter¬ 
ested  in  how  people  of  varying  levels  of  expertise  designed  experi¬ 
ments.  Next,  subjects  were  given  the  problem  statement  together  with 
tha  talk  aloud  instructions.  After  subjects  had  read  the  problem 
statement,  a  cassette  recorder  was  started  which  recorded  the  sub¬ 
jects'  verbalizations.  Subjects  were  allowed  to  use  paper  and  pencil 
If  they  wished  to  do  so.  Only  two  of  ths  design  experts  made  use  of 
these  materials.  The  subjects  themsalvss  Indicated  when  they  thought 
they  had  solved  the  problem. 
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2.3  Predictions 

Based  on  our  took  analysis ,  cha  following  predictions  are  made. 

First,  strategies  for  pursuing  a  paradigm  will  need  Co  accomplish  the 
goal  of  controlling  variance.  For  novel  problems,  experienced  re¬ 
searchers  will  use  their  knowledge  of  design  principles  in  order  to 
achieve  control  of  variance.  Hence,  there  will  be  more  statements  In 
the  "Select  design  principles"  category  for  the  design  experts  than 
for  the  beginners  and  the  domain  experts.  Second,  design  experts  will 
switch  more  often  between  the  categories  "Select  design  principles" 
and  "Pursue  paradigm"  than  beginners  and  domain  experts.  Note  that 
both  beginners  and  domain  experts  also  need  to  accomplish  the  goal  of 
controlling  variance.  However,  compared  with  design  experts,  there 
will  be  fewer  statements  in  this  category  for  these  two  groups.  Begin¬ 
ners  will  have  problems  retrieving  design  principles,  and  domain 
experts  will  incorporate  these  principles  directly  into  their  designs, 
without  mentioning  them  explicitly.  Intermediates  will  perform  in 
between  the  beginners  and  the  design  experts. 

Second,  overall,  domain  experts  will  switch  fewer  times  between  cat¬ 
egories  than  design  experts  and  novices,  because  the  domain  experts 
encounter  fewer  impasses  than  the  two  other  groups.  However,  the 
design  experts  will  conform  more  to  the  expert  model  than  the  begin¬ 
ners  and  the  Intermediates,  because  of  their  more  abstract  problem 
conception  schema  and  because  of  their  use  of  domain- dependent  stra¬ 
tegic  knowledge. 

Third,  paradigms  are  knowledge  structures  that  are  deposited  In  a 
working  memory  with  a  limited  capacity.  Paradigms  will  therefore  be 
successively  refined,  using  a  strategy  of  progressive  deepening.  Both 
beginners  and  domain  experts  will  not  use  the  strategy  of  progressive 
deepening.  The  beginners'  knowledge  Is  insufficient  for  successively 
adding  new  Information  to  working  memory.  The  domain  experts  will, 
once  they  have  chosen  a  particular  paradigm,  pursue  that  paradigm 
without  having  to  search  for  design  principles  and  without  having  to 
reread  the  problem  statement.  The  domain  experts  will  therefore  not 
need  to  go  over  the  same  paradigm  again  and  again.  The  Intermediates 
will  probably  have  developed  rudimentary  paradigms,  but  It  Is  unclear 
Whether  they  will  use  progressive  deepening. 
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3  RESULTS  AND  DISCUSSION 

The  results  (action  la  atructurad  aa  followa.  I  will  atart  with  some 
summary  atatlatlca  on  tha  numb a r  of  atatamanta  In  aach  category  of  tha 
coding  aystem,  tha  total  problem  aolvlng  time,  and  tha  total  number  of 
aolutlona.  These  raaulta  give  an  overview  of  aome  groaa  dlfferancaa 
among  tha  groups.  Tha  theoretical  framework  will  provide  tha  categ- 
orlaa  for  diacuaalng  tha  other  raaulta.  Mora  specifically,  tha  follow¬ 
ing  element*  will  be  dlacuaaed:  goal  atructura,  atrategiaa  for  goal 
attainment  and  lmpaaae  recovery,  and  problem  conception  achama. 


3.1  Summary  atatlatlca 

Table  I  ahows  the  total  number  of  atatementa  In  the  protocola  (with 
the  exclusion  of  monitoring  statements),  the  total  problem  solving 
time  for  the  four  groupa  of  subjects,  and  the  total  number  of  sol¬ 
utions  (paradigms)  mentioned  by  subjects. 

Table  I  Average  total  number  of  statements  in  proto¬ 
cols,  average  total  problem  solving  time  (In  minutes) 
for  the  four  groupa  of  subjects,  and  average  number  of 
solutions . 


number 

time 

solutions 

Beginners 

27 

5 

1.0 

Intermediates 

60 

9 

2.0 

Design  Experts 

66 

13 

3.0 

Domain  Experts 

68 

14 

4.2 

Clearly,  experts  came  up  with  more  solutions;  hence,  they  took  much 
longer  to  solve  tha  problem  and  generated  more  verbal  statements  than 
beginners . 

Table  II  shows  the  number  of  statements  and  the  proportion  (In 
brackets)  In  each  category  of  the  coding  scheme. 
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Table  11  Average  number  of  atatementa  and  proportion  In 
each  category  of  the  coding  scheme  for  the  four  groupa. 


Bag 

Int 

DesExp 

DomExp 

Orientate  on  task 

1 

(31) 

i 

(2%) 

3  (3%) 

0 

(0%) 

Understand  problem 

5 

(17%) 

9 

(14%) 

10  (15%) 

19 

(28%) 

Select  paradigm/analogy 

3 

(10%) 

7 

(11%) 

12  (18%) 

17 

(25%) 

Select  design  principles 

6 

(20%) 

13 

(20%) 

15  (22%) 

5 

(7%) 

Pursue  paradigm 

12 

(40%) 

30 

(47%) 

25  (371) 

27 

(40%) 

Evaluate  teak 

0 

(0%) 

0 

(0%) 

1  d%) 

0 

(0%) 

Monitoring 

3 

(10%) 

4 

(6%) 

1  d%) 

1 

(0.5%) 

Since  subjects  generated  more  statements  with  Increasing  expertise, 
the  analysis  on  differences  between  categories  was  carried  out  on  the 
proportion  of  statements  within  each  category.  A  Kruskal -Wallis  Analy¬ 
sis  of  Variance  with  level  of  expertise  as  grouping  variable  and  the 
proportion  of  statements  as  dependent  variable  shoved  a  marginally 
significant  difference  between  the  four  groupa  for  the  category  Select 
design  principles  (T-6.40,  p-0,09).  The  remaining  categories  were  not 
significantly  different  for  the  four  groupa.  The  first  prediction  Is 
therefore  partly  confirmed:  the  design  experts  used,  across  the  whole 
protocol,  more  design  principles  than  domain  experts.  Contrary  to  what 
was  predicted,  the  beginners  made  as  much  use  of  design  principles  as 
the  design  experts,  when  the  total  number  of  statements  generated  are 
controlled  for. 


3.2  Coal  structure 

In  order  to  detect  an  ordering  in  the  goals  subjects  successively 
pursued,  the  switches  between  the  different  categories  in  the  proto¬ 
cols  were  counted.  To  determine  the  nature  of  the  switches  between  the 
different  categories,  the  three  categories:  Orientate  on  task,  Evalu¬ 
ate  task  and  monitoring  were  excluded  from  further  analysis.  The 
reason  for  the  exclusion  was  that  these  three  categories  are  not  part 
of  the  goal  structure  of  interest  in  this  study.  Hence,  there  were 
four  categories  left:  Understand  problem  (U),  Select  paradigm/analogy 
(SP),  Pursue  paradigm  (PP),  and  Select  design  principles  (DP). 

The  switches  between  the  Individual  statements  were  classified  and 
counted  for  each  subject.  The  number  of  switches  was  nsxt  added  for 
all  subjects  within  one  group.  The  switches  between  categories  were 
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tasted  both  against  a  quasi -random  modal  and  against  an  "expert 
model”,  in  order  to  detect  whether  the  data  significantly  differed 
from  these  models.  A  test  against  two  models  gives  more  confidence  in 
the  general  pattern  of  results  when,  as  predicted,  one  model  is  ac¬ 
cepted  and  the  other  rejected.  In  this  case,  the  random  model,  but  not 
the  expert  model,  would  fit  the  data  of  the  beginners  well,  while  Che 
reverse  pattern  is  predicted  for  the  expert  groups. 

The  diagonal  was  excluded  from  these  analyses,  because  the  Interest  in 
this  study  was  not  primarily  in  how  long  subjects  would  stay  in  one 
category.  Before  presenting  the  results  of  the  model  testing,  both 
models  will  be  discussed  in  more  detail  below. 

The  quasi-random  model  takes  into  account  the  number  of  items  in  a 
particular  category  and  determines  the  likelihood  of  going  from  a 
particular  category  to  another  category.  Therefore,  the  different 
number  of  switches  between  the  different  groups  of  subjects  is  con¬ 
trolled  for.  If  there  are  more  items  in  a  particular  category,  then 
chances  are  higher  that  a  transition  will  be  made  to  that  category, 
irrespective  of  the  current  category. 

The  expert  model  la  shown  In  Figure  2. 


Fig.  2  Expert  model. 
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The  expert  modal  only  allova  switches  batwaan  lmmadiataly  preceding 
and  lmmadiataly  following  categories.  Thla  ylalda  cha  following  pat* 
tarn  of  'lagal'  (L)  and  'Illegal*  (1)  awltchaa. 


Tabla  III  'Lagal'  and  'lllagal*  awltchaa  according  to 
the  'axpart  modal*. 


to 

from 

u 

SP 

PP 

DP 

u 

. 

L 

I 

I 

SP 

L 

- 

L 

I 

PP 

I 

L 

- 

L 

DP 

I 

I 

L 

- 

A  constant  error  parameter  was  Included  for  every  ’Illegal'  transi¬ 
tion.  Thus,  every  Illegal  awltch  was  consldared  equally  likely.  The 
parameters  In  the  model  correspond  to  weights  attached  to  the  cat¬ 
egories.  The  chance  of  going  from  one  category  to  the  other  Is  propor¬ 
tional  with  the  (relative)  weight  of  the  category.  There  were  three 
parameters  In  the  model  that  had  to  be  estimated:  the  error  parameter 
and  the  parameters  corraaponding  to  switches  from  SP  to  U  and  from  PP 
to  SP.  All  other  parameters  could  be  derived  from  these  three  parame¬ 
ters.  Two  factors  are  Important  when  testing  the  data  against  the 
expert  model: 

1)  the  'fit*,  expressed  in  a  chi-square  measure; 

2)  the  magnitude  of  the  error  parameter,  relative  to  the  other 
parameters . 

Both  factors  are  important,  since  It  is  theoretically  possible  to  have 
a  good  fit  and  a  high  value  for  the  error  parameter  at  the  same  time. 
Thla  would  be  the  case  when  the  Illegal  transitions  would  all  be  equal 
In  magnitude  and  relatively  high  at  the  same  time.  The  predictions 
were  that,  for  the  expert  groups,  first,  the  data  would  not  signifi¬ 
cantly  deviate  from  the  expert  modal,  and  secondly,  the  error  parame¬ 
ter  would  be  low  compared  with  the  other  parameters.  The  value  of  the 
error  parameter  was  therefore  divided  by  the  average  value  of  the 
other  parameters. 

The  parameters  In  the  models  are  estimated  by  minimizing  a  chi-square 
function.  Hence,  the  predicted  and  observed  frequencies  of  switches 
occurring  In  the  protocols  are  compared  and  expressed  In  a  chi-square 
measure.  Table  IV  ahows  the  results  of  the  parameter  estimation. 
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Table  IV  Chi-squares  (df-5)  for  eh*  parameter  astlmacas 
of  Che  four  groups . 


Random  model 

Expert  model 

Beginners 

6.14 

(N.S.) 

Intermediates 

20.42 

(p<0.001) 

11.72 

(p<0.05) 

Design  Experts 

33.98 

(p<0.001) 

9.28 

(N.S.) 

Domain  Experts 

21.60 

(p<0.001) 

4.56 

(N.S.) 

The  pattern  of  swltchea  between  categoriea  for  the  beglnnere  did  not 
significantly  devlata  from  the  quasi-random  model.  Both  expert  groups 
and  the  intermediates  did  significantly  deviate  from  the  quasi-random 
model. 

Since  the  beginners  did  not  significantly  differ  from  the  quasi-random 
model,  there  was  no  need  to  test  their  data  against  the  expert  model. 
If  one  would  do  so,  the  error  parameter  would  be  too  high  relative  to 
the  other  parameters.  The  intermediates  significantly  deviated  from 
the  expert  model.  The  chi-square  values  for  the  experts  were  not 
significant.  The  legal  parameters  were,  on  average,  five  times  as  high 
as  the  illegal  parameters.  The  estimated  value  for  the  error  parameter 
was  0.15  for  the  Intermediates  and  0.13  for  the  expert  groups.  These 
values  are  very  acceptable.  Therefore,  the  conclusion  is  that  the 
transition  data  for  both  groups  of  experts  can  be  fitted  with  the 
'expert  model'  described  above.  The  intermediates'  data  could  not  be 
fitted  with  both  models. 

In  order  to  test  the  second  prediction,  namely  that  domain  experts 
would  switch  fewer  times  between  categories  than  the  other  groups,  the 
number  of  switches  between  categories  was  divided  by  the  total  number 
of  switches.  Percentages  were  calculated  since  the  protocols  of  the 
four  groups  were  of  unequal  length.  The  percentages  for  the  four 
groups  are  shown  in  Table  V. 

Table  V  Percentage  of  switches  between  categories  for 

the  four  groups. 


beginners  38l 
intermediates  42% 
design  experts  45% 
domain  experts  20% 
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The  proportion  of  switches  bo tween  categories  ie  ouch  lower  for  the 
domain  expert*  than  for  the  other  group*.  The  difference  between  the 
four  group*  1*  significant  (Kruskal -Wallis  T-5.91,  p-0.05).  This 

confirms  our  second  prediction. 

In  summary,  the  results  concerning  the  goal  structure  yield  the  fol¬ 
lowing  pattern: 

-  goth  group*  of  experts  switched  between  goals  according  to  the 
expert  model 

-  The  domain  experts  did  not  switch  as  often  between  goals  as  the 
other  groups. 

The  statistics  discussed  above  have  given  an  overall  picture  of  some 
salient  differences  between  groups  in  terms  of  the  goal  structure.  The 
next  section  will  describe  the  strategies  used  by  the  different  groups 
of  subjects  to  accomplish  their  goals  or  to  recover  from  impasses.  The 
focus  will  be  on  the  design  experts,  the  other  groups  serving  primar¬ 
ily  as  controls. 


3.3  Strategies 

Design  experts  may  use  deliberate  strategies  whenever  they  encounter 
an  Impasse  and  they  have  to  switch  to  another  problem  space  (cat¬ 
egory)  .  The  following  sections  will  first  describe  these  strategies , 
Illustrating  them  with  protocol  fragments  where  necessary,  than  de¬ 
scribe  the  criteria  used  to  determine  the  use  of  a  particular  strat¬ 
egy,  and  finally  describe  the  results  in  terms  of  these  strategies. 

3.3.1  Description  of  strategies 
Strategy  l:  Hypothetical  reasoning 

Hypothetical  reasoning  Is  a  strategy  that  Is  used  when  the  goal  Is  to 
select  a  paradigm  and  there  Is  Insufficient  knowledge  to  choose  among 
paradigms.  This  stratsgy  consists  of  determining  the  likely  outcome  of 
a  particular  paradigm  and  comparing  this  outcome  with  what  Is  asked 
for  in  the  research  question.  The  reasoning  process  is  called  'hypo¬ 
thetical'  because  the  search  for  a  paradigm  is  carried  out  in  a  prob¬ 
lem  space  in  which  various  alternatives  are  considered  as  hypotheses 
and  are  evaluated  before  they  are  actually  Implemented.  Hypothetical 
reasoning  Is  a  fora  of  planning,  because  the  stratsgy  is  applied  to  an 
abstract  search  space,  in  which  only  the  outcomes  of  paradigms  are 
represented  and  all  other  datalls  ara  Ignored. 
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Design  Expert  1  deliberately  used  this  strategy  end  wee  aware  of  Its 
usefulness,  as  witnessed  by  the  following  protocol  statements: 

Well,  suppose  you  have  done  an  experiment  like  that,  at 
least  that  is  always  my  approach,  what  do  you  have?  When 
you  have  those  data,  what  can  you  do  with  them?  If  you 
don't  know,  O.K.  I  have  collected  data,  but  you  don't 
know  exactly  what  to  do  with  those  data,  then  perhaps 
you  should  not  do  the  experiment  at  all. 

Strategy  2:  Mental  alnulatlon 

The  strategy  of  mental  simulation  makes  use  of  the  fact  that  design 
experts  have  represented  paradigms  as  scenarios.  When  subjects  tried 
to  fill  in  the  details  of  a  particular  paradigm,  they  would  imagine 
how  the  experimental  procedure  would  look  like.  Imagining  the  pro¬ 
cedure  often  suggested  extra  information  to  be  Included  in  the  para¬ 
digm,  or  difficulties  that  had  to  be  resolved.  The  difficulties  arose 
because  particular  design  decisions  violated  certain  validity  issuea, 
as  specified  by  certain  design  principles. 

An  example  where  a  problem  is  noted  when  mentally  simulating  the 
procedure,  is  the  following  from  Design  Expert  3's  protocol: 

Now  you  have  the  problem  of:  you  have  three  stimuli,  you 
have  a  subject,  you  have  all  controls,  and  what  are  you 
going  to  do  then?  ...  Well,  I  think  three  stimuli  are 
not  enough,  so  you  could  think  about  constructing  a 
perceptual  apace  in  which  you  compare  those  colas  with 
the  larger  group  of  soft  drinks. 

The  following  example  from  Design  Expert  l's  protocol  illustrates  the 
use  of  general  design  principles: 

And  then,  secondly,  the  subject  gets  a  drink,  and  than  I 
do  not  know  enough  about  details,  whether  you  have  to 
eat  a  little  bit  of  bread  after  that,  or  wait  a  minute, 
or  drink  something  neutral  in  between,  I  am  not  an 
expert  In  that  area. 

In  the  quote  above,  the  subject  interrupts  the  filling  in  of  the 
details  of  the  paradigm  (after  "gets  a  drink,  . when  he  realizes 
that  the  subjeet  gets  another  drink  after  the  first  one  and  that  the 
taste  of  these  two  drinks  may  influence  one  another.  One  general 
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design  principle  Is  to  aske  sure  thst  the  measuring  Instrument  does 
not  change  over  the  course  of  the  experiment  (cf.  Cook  &  Campbell, 
1979,  p.52).  In  this  case,  the  measuring  Instrument  Is  the  human 
taster.  Design  Expert  1  comes  up  with  several  ways  of  preventing  this 
threat  to  the  Internal  validity  of  the  design,  but  does  not  choose 
among  one  of  them  on  the  ground  that  he  is  not  an  expert  when  It  comes 
to  sensory  psychology. 

3.3.2  Criteria  for  Identification  of  strategies 
Strategy  1:  Hypothetical  reasoning 

The  strategy  of  hypothetical  reasoning  occurs  before  a  paradigm  is 
pursued.  Subjects  tentatively  <.-ci)uate  various  paradigms  before  choos* 
lng  one.  Evidence  in  the  protocols  for  this  strategy  comes  from  the 
frequent  use  of  words  such  as  "suppose"  and  "would  do". 

Strategy  2:  Hental  simulation 

Evidence  for  the  strategy  of  mental  simulation  comes  from  sequences 
such  as  'first... and  then. ..and  after  that',  For  instance:  "And  then 
you  make  a  list  (...)  with  a  number  of  dimensions,  and  then  group 
those  dimensions  (...)  and  then  let  them  fill  them  In  (...)  and  then 
take  those  kinds  of  scores”.  Just  using  the  words  "and  then  ...  and 
then"  is  not  evidence  per  se  for  the  use  of  mental  simulation.  It 
might  as  well  be  evidence  for  just  summing  up  the  steps  in  an  already 
stored  plan.  Mental  simulation,  on  the  other  hand,  means  trying  out 
alternatives  with  the  possibility  of  being  corrected.  I  will  restrict 
the  definition  of  mental  simulation  therefore  to  those  cases  where 
pursuing  a  paradigm  (Indicated  by  the  words  "and  then  ...  and  then") 
Is  Interrupted  by  selecting  a  design  principle. 

3.3.3  Description  of  results  In  tens  of  strategies 
Strategy  1:  Hypothetical  reasoning 

Design  Expert  1  was  the  only  subject  who  used  this  strategy.  The  other 
design  experts  immediately  chose  for  a  particular  paradigm,  without 
extensively  evaluating  them  against  other  paradigms.  The  two  basic 
paradigms  Design  Expert  1  came  up  with  ware: 

A;  pairwise  comparisons  of  colas 
B:  tasting  one  cola  after  the  other 

There  were  two  versions  of  both  paradigm  A  and  B,  and  the  major  task 
of  the  subject  was  to  chooss  between  those  versions.  The  two  versions 
are  referred  to  as  Al  and  A2 ,  and  B1  and  B2 . 
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An  Interpretation  of  the  protocol  of  Design  Export  1,  togathor  with 
impasses  and  rapalra,  appaara  In  Appandlx  B.  Appendix  B  shove  that,  by 
using  this  strategy,  Design  Expert  1  was  able  to  eliminate  two  para¬ 
digms  from  his  list  and  ended  up  by  positively  evaluating  paradigm  A2. 
This  paradigm  was  subsequently  pursued.  The  strategy  of  hypothetical 
reasoning  was  used  from  the  fourth  to  the  eleventh  minute  In  the 
protocol.  This  constitutes  50%  of  the  total  problem  solving  time.  The 
remainder  of  the  time  was  taken  up  by  understanding  the  problem  and 
pursuing  the  paradigm.  In  conclusion,  the  strategy  of  hypothetical 
reasoning  enabled  Design  Expert  1  to  constrain  his  search  for  possible 
paradigms . 

Strategy  2:  Mental  simulation 

The  protocols  of  all  subjects  were  scored  for  the  use  of  mental  simu¬ 
lation  as  defined  above.  The  average  frequency  of  use  in  the  four 
groups  of  subjects  is  shown  in  Table  VI. 


Table  VI  Average  frequency  of  use  of  the  strategy  of 
mental  simulation  for  the  four  groups  of  subjects. 


beginners 

0.5 

Intermediates 

3.0 

design  experts 

3.3 

domain  experts 

0.5 

The  Intermediates  and  the  design  experts  made  use  of  the  strategy  of 
mental  simulation  six  times  as  often  as  the  beginners  and  the  domain 
experts,  A  Chi-square  test  on  the  total  frequency  of  use  of  the  strat¬ 
egy  showed  a  significant  difference  between  the  four  groups,  Chi- 
square  (3)-12.  38,  p-0.006.  Note  that  the  four  groups  of  subjects  are 
made  comparable  to  each  other  by  carrying  out  e  Chi-square  test  on  the 
total  number  of  statements  In  the  protocols.  The  Chi-square  test  then 
uses  the  relative  frequencies  for  the  four  groups,  thus  controlling 
for  any  differences  between  the  four  groups  In  the  total  number  of 
statements  verbalized. 

The  results  above  have  already  shown  that,  as  predicted,  the  design 
experts  used  mors  design  principles  than  domain  experts.  The  second 
part  of  this  prediction  statad  that  design  experts  would  switch  more 
between  pursuing  a  paradigm  and  use  of  design  principles,  and  vice 
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versa,  chan  cha  beginners  and  tha  domain  axparta.  Tha  average  numbar 
of  awlcchea  batwaen  chaae  two  categories  la  ahown  In  Table  VII. 


Table  VII  Avaraga  number  of  avitchaa  batwaen  catagorlaa 
"Pursue  paradigm"  and  "Select  dealgn  principle"  (and 
vice  varaa)  for  tha  four  groupa  of  aubjacta. 


Beginners 

3.3 

Intermediates 

13.7 

Design  experts 

14.7 

Domain  exparts 

3.4 

Tha  difference  between  tha  four  groupa  waa  highly  algnlf leant,  aa 
indicated  by  a  Chi -square  teat  on  tha  total  frequency  of  switches 
between  the  categories,  Chi - square (3)-46. 48,  p<0.001.  Clearly,  the 
design  experts  and  the  Intermediates  switched  more  often  between  the 
two  categories  than  the  other  groupa.  Hence,  our  prediction  is  con¬ 
firmed. 

In  conclusion,  the  design  experts  and  the  intermediates  frequently 
switched  between  pursuing  a  paradigm  and  selecting  and  applying  a 
general  design  principle  to  that  paradigm.  Both  groups  mentally  simu¬ 
lated  the  experimental  procedure  and  frequently  interrupted  their 
problem  aolvlng  whenever  a  violation  of  a  general  design  principle  was 
noted.  The  atrategy  of  mental  simulation  was  lass  frequently  used  by 
the  beginners  and  the  domain  experts. 

The  next  section  will  describe  the  results  concerning  the  final  el¬ 
ement  in  our  theoretical  framework,  the  problem  conception  schema. 


3.4  Problem  conception  schema 

The  third  prediction  stated  that  design  experts  would  use  the  general 
elementa  in  their  problem  conception  schema  when  they  adapted  the 
schema  to  the  problem,  and  that  they  would  uae  a  progressive  deepening 
strategy.  First,  evidence  will  be  shown  for  the  use  of  general  el¬ 
ements  in  the  problem  conception  schema.  Second,  evidence  for  a  pro¬ 
gressive  deepening  atrategy  will  be  discussed. 
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The  schematising  affect  of  Che  problem  conception  on  the  ill-struc¬ 
tured  problem  preeenced  Co  Che  subject!  mey  be  evident  from  the  fol¬ 
lowing  elements  In  the  protocols: 

•  the  problem,  the  research  question,  or  the  experiment  are  categor¬ 
ized,  e.g.,  "a  problem  on  taste”,  or  "consumer  research” 

•  missing  information  la  supplied;  this  will  apply  particularly  to 
the  Important  points  in  the  problem  description,  l.e.,  the  causa  of 
the  complaints  and  the  correctness  of  the  manufacturer's  research 
question 

-  details  are  abstractad  from  or  changed;  the  details  concern  the 
other  cola  brands,  the  broad  definition  of  taste,  the  bureau  for 
market  research,  and  the  target  population 

-  attention  is  directed  to  the  key  elements  in  the  problem  formulati¬ 
on,  l.e.,  the  phrase  "what  people  taste  exactly". 

As  predicted,  the  design  experts  used  their  problem  conception  schema 
for  structuring  the  ill-structured  problem  they  were  confronted  with. 
The  four  elements  mentioned  above  will  first  be  illustrated  with 
relevant  protocol  segments  from  the  design  experts'  protocols  before 
turning  to  a  quantitative  analysis  across  the  four  groups  of  subjects. 
All  four  elements  should  be  clearly  identifiable  in  the  protocols  as 
resulting  from  a  particular  problem  conception  schema,  rather  than 
being  Isolated  elements  that  subjects  derive  from  their  general  world 
knowledge . 

3.4.1  Categorization 

Paradigms  were  chosen  quickly  on  Che  basis  of  a  structural  similarity 
between  the  perceived  problem  and  a  particular  paradigm.  Design  Expert 
2  did  not  evaluate  various  paradigms  against  each  other  as  extensively 
as  Design  Expert  1.  One  of  his  first  statements  was: 

I  understand  that  a  kind  of  constraint  is  that  you  are 
thinking  of  a  panel  experiment. 

This  statement  indicates  that  Design  Expert  2  had  quickly  selected  a 
paradigm  and  saw  it  as  his  main  task  to  pursue  this  paradigm.  Design 
Expert  3  reformulated  the  problem  as  follows: 

He  wants  to  know  what  people  taste  exactly,  so  what  they 
take  to  be  the  taste  of  cola.  That  is  of  course  a  pretty 
vague  concept  (...)  So  what  you  want  to  measure  exactly 
is:  where  do  ay  colas  fit  into  a  kind  of  perceptual 
space  of  soft  drinks. 
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Design  Expect  3  wee  familiar  with  Multidimensional  Seeling  technique*, 
alnce  he  had  recently  conducted  some  experiments  on  the  perception  of 
highways  using  the  Personal  Construct  method  and  rating  scales.  It  may 
very  well  be  that  reading  about  perception  of  taste  of  colas  immedi¬ 
ately  suggested  the  same  paradigm  to  him.  This  suggestion  is  an  exam¬ 
ple  of  analogical  reasoning  in  which  the  target  problem  and  its  ana¬ 
logue  share  structural  features  (the  abstract  concept  of  "perceptual 
space")  but  are  superficially  dissimilar  (colas  versus  highways). 
Design  Expert  2  was  also  aware  of  this  structural  similarity,  alnce  he 
remarked,  when  trying  to  come  up  with  a  third  paradigm: 

Yes,  X.  [Design  Expert  3]  has  once  used  a  technique, 
that  perception  of  highways,  X  would  talk  to  X.  how  he 
did  that.  Because  I  think  the  problem  is  very  similar. 

Thus,  the  research  question  was  categorized  abstractly  by  all  design 
experts  as  falling  in  the  general  category  of  "perceptual  experi¬ 
ments"  ,  in  which  an  underlying  space  of  dimensions  is  identified  by 
means  of  distance  ratings  between  stimuli. 

3.4.2  Supplying  missing  information 

During  the  first  minute  of  his  protocol,  Design  Expert  1  remarked: 

So  what  they  taste  exactly,  that  really  is  his  question. 

But  that  does  not  mean  that  you  have  to  take  that  seri¬ 
ously  as  a  researcher,  because  what  does  the  manufac¬ 
turer  know.  Perhaps  it  is  also  important  to  know  whether 
they  are  able  to  taste  any  differences  at  all.  And  then 
pairwise  comparisons  may  be  useful. 

This  fragment  shows  the  use  of  a  particular  paradigm,  pairwise  com¬ 
parisons,  in  refining  the  research  question. 

3.4.3  Abstraction  and  changing  of  details 

The  following  fragment  from  Design  Expert  3’s  protocol,  already  dis¬ 
cussed  above  in  the  context  of  mental  simulation,  shows  that  the 
number  of  stimuli  is  enlarged,  because  the  subject  considers  using  a 
multi-dimensional  scaling  paradigm.  In  this  paradigm,  a  space  of 
underlying  dimensions  is  constructed,  using  a  number  of  stimuli  that 
is  considerably  larger  than  the  number  of  dimensions  extracted. 

Now  you  have  the  problem  of:  you  have  three  stimuli,  you 
have  a  subject,  you  have  all  controls,  and  what  ara  you 
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going  Co  do  chon?  ...  Wall,  I  chink  chraa  stimuli  ara 
noc  anough,  ao  you  could  chink  about  conatructing  a 
perceptual  apace  in  which  you  compare  thoae  colaa  with 
Che  larger  group  of  coft  drink*. 

Design  Expert  1  also  enlarged  Che  number  of  stimuli  when  considering  a 
multi -dimensional  scaling  experiment. 

3.4.4  Attention  focused  on  kay  elements 
Design  Expert  2  started  by  saying: 

If  the  question  is  really  what  they  taste  exactly,  than 
I  think  you  have  to  use  panel  research. 

This  fragment  clearly  shows  that  a  key  element  in  the  problem  descrip¬ 
tion  triggers  a  particular  paradigm.  The  same  was  shown  above  under 
the  heading  "supplying  missing  information" ,  where  Design  Expert  1 
retrieved  a  pairwise  comparisons  experiment  after  having  read  the 
phrase  "what  people  taste  exactly". 

The  beginners  invariably  immediately  translated  "taste"  into  a  par¬ 
ticular  dependent  measure,  e.g.,  a  rating  scale  or  a  questionnaire. 
There  was  no  evidence  for  a  categorization  of  the  problem  or  the 
experiment,  l.e. ,  the  dependent  measure  they  selected  was  not  part  of 
a  larger  conceptual  structure,  but  functioned  es  e  goal  by  itself. 
Hence,  their  choice  for  a  dependent  measure  was  based  on  superficial 
rather  than  structural  features  in  the  problem  statement.  One  of  these 
superficial  features  was  "the  taste  of  Coca  Cola".  Reading  about  the 
taste  of  Coca  Cola,  a  lot  of  beginners  were  reminded  of  the  "Pepsi 
challenge",  that  had  been  shown  on  television  as  a  commercial  recent¬ 
ly.  Mote  that  this  analogy  is  actually  misleading,  since  the  Pepsi 
challenge  is  about  preferences  for  a  certain  brand  of  cola,  whereas 
the  research  question  is  about  "what  people  really  taste",  which  is  a 
descriptive  rather  than  a  hedonic  question.  Beginners  therefor*  fre¬ 
quently  misinterpreted  the  research  question.  Interestingly,  the 
beginners  could  frequently  bring  to  boar  a  lot  of  potentially  relevant 
knowledge  about  soft  drinks,  s.g.,  the  importance  of  the  image  of  soft 
drinks.  They  failed  to  incorporate  this  knowledge  into  an  overall 
problem  conception  schoma,  because  they  lacked  such  a  schema.  There¬ 
fore,  beginners  also  did  not  supply  missing  information,  abstract  from 
details,  or  focused  their  attention  on  the  key  elements  in  the  problem 
formulation. 
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The  Intermediates  did  not  categorize  tho  problem  In  the  aana  abatraet 
way  aa  tha  daalgn  axparta.  Lika  tha  baglnnara,  tha  lntarmadlataa 
atartad  by  selecting  a  particular  dapandant  measure,  baaad  upon  tho 
fragment  "what  paopla  taata" .  Tho  lntamadlataa  differed  from  tho 
baglnnara,  howavar,  In  that  tha  dapandant  maaaura  thay  choaa  waa  part 
of  a  paradigm,  aueh  aa  multl-dlmanaional  acallng  or  aelf-report  ques- 
tlonnalraa.  Alao  Ilka  tha  baglnnara,  and  unllka  tho  daalgn  axparta, 
thay  did  not  avaluata  their  dealgna  agalnat  an  abatraet  goal.  Two  of 
tha  lntarmadlataa  Incorrectly  apeclflad  tha  goal  aa:  "how  wall  doaa 
Coca  Cola  taata".  Tha  lntarmadlataa  did  not  check  whether  tha  raaulta 
of  thalr  experiment*  would  be  of  any  uaa  to  tha  cola  manufacturer.  The 
daalgn  axparta  alwaya  checked  tha  uaa  of  thalr  raaulta. 

Tha  domain  exports'  choice  of  paradigm  waa  based  on  a  thorough  analy¬ 
sis  of  tha  problem  statement.  Tha  thorough  problem  analysis  la  shown 
by  tha  following  quantitative  result  on  tha  number  of  statements  In 
tha  Understand  Problem  category  that  occurred  before  the  subjects 
actually  pursued  a  paradigm. 


Table  VIII  Average  number  of  statements  In  Understand 
problem  category  that  occurred  before  the  first  state¬ 
ment  in  the  Pursue  paradigm  category. 


Beginners 

2.2 

Intermediates 

6.3 

Design  Experts 

5.7 

Domain  Experts 

10.5 

A  Kruskal -Wallis  Analysis  of  Variance  with  level  of  expertise  as 
grouping  variable  and  the  number  of  statomonts  aa  dependent  variable 
showed  a  significant  difference  between  tho  four  groups  for  tha  Under¬ 
stand  category  (T-9.43,  p-0.02).  This  result  Indicates  that  tha  domain 
axparta  devoted  more  attention  to  tha  problem  statement  before  pursu¬ 
ing  a  particular  paradigm  than  tho  other  groups.  Tha  different  number 
of  solutions  different  groups  of  subjects  came  up  with  is  not  an  issue 
hero,  since  tha  analysts  is  carried  out  on  tha  statements  before  the 
first  solution  is  mentioned. 

Domain  Expert  1  spent  a  great  deal  of  time  analyzing  tho  problem 
statement.  He  started  his  protocol  by  saying: 
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Did  the  manufacturer  translate  the  problem  In  the  right 
way?  (...)  The  question  la  whether  thoae  complaint# 
concerning  taate  do  indeed  concern  the  teate.  You  can 
have  your  doubta  about  that.  (...)  Look,  a  complaint,  a 
remark:  there  are  complalnta,  that  aaka  for:  how  la  the 
altuation,  where  do  thoae  complalnta  preclaely  come 
from.  Becauae  1  don't  think  it  la  right  to  directly  do 
aenaory  research. 

Domain  Expert  1  then  went  on  to  enumerate  poaaibla  cauaea  for  the  com- 
plainte  about  the  taate  of  Coca  Cola:  a  change  in  raw  material# , 
natural  variations  in  raw  materials,  a  fault  in  the  process,  residues 
of  detergents  In  cola  bottles,  fault  with  the  Internal  quality  con¬ 
trol,  poor  marketing  and  advertisement.  Domain  Expert  2  mentioned  some 
other  possible  causes:  Coca  Cola  may  have  become  too  expenalve,  the 
bottles  may  have  changed  In  appearance,  the  "cola-generation"  Is 
getting  old  and  switches  to  other  drinks,  perhaps  due  to  the  Introduc¬ 
tion  of  "light  bears".  These  two  domain  experts  generated  a  large 
number  of  hypotheses  that  might  be  responsible  for  the  complaints. 
Most  of  these  hypotheses  do  not  require  sensory  research,  since  the 
problem  is  not  necessarily  caused  by  the  taste  of  cola  as  such.  How¬ 
ever,  both  domain  experts  went  on  and  assumed  that  the  problem  was 
Indeed  a  sensory  problem.  From  this  point  on,  their  problem  solving 
was  very  similar  to  that  of  the  other  two  domain  experts,  and  con¬ 
sisted  of  retrieving  standard  senaory  paradigms  from  LTM.  The  other 
two  domain  experts  spent  much  less  time  analyzing  the  problem  state¬ 
ment,  because  they  assumed  that  some  form  of  sensory  research  had  to 
be  carried  out.  This  assumption  was  not  made,  however,  without  ex¬ 
plicitly  questioning  the  manufacturer's  research  question.  Domain 
Expert  3  said: 

The  first  thing  I  would  do  is  talk  to  Coca  Cola  and  ask 
If  they  really  mean  what  they  ask.  If  they  really  want 
to  know  what  people  taste  exactly,  then  they  can  never 
do  that  with  market  research.  Then  you  have  to  use  much 
more  complicated  methods,  and  then  I  would  advise  them 
to  set  up  a  descriptive  panel. 

Besides  supplying  missing  information,  domain  experts  often  criticized 
and  changed  details  in  the  problem  description  (e.g.,  use  of  the 
bureau  for  market  research  was  deemed  unnecessary,  the  target  popula¬ 
tion  was  defined  too  broadly,  use  of  the  house  brand  of  cola  was  found 
illogical).  When  pursuing  a  particular  paradigm,  domain  experts  often 
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did  not  refer  to  cola  at  all,  but  rathar  dascrlbad  general  tachnlquaa 
applicable  In  all  klnda  of  sensory  raaearch.  Thla  finding  indicate* 
that  detail*  ware  abatraotad  from  in  tha  problem  conception  schema. 

In  concluaion,  only  tha  disign  expert*  and  the  domain  export*  uaad  the 
ganaral,  structural,  a lament*  in  thalr  problaa  conception  achama.  Use 
of  those  ganaral  element*  allowed  them  to  structure  the  ill-structured 
problaa  they  were  confronted  with.  Tha  beginners  and  the  intermediate* 
uaad  superficial  feature*  in  tha  problem  description  to  choose  an 
analogy,  in  tha  case  of  tha  beginners,  or  a  more  general  paradigm,  in 
the  case  of  the  intermediate*.  Therefore,  the  first  part  of  the  third 
prediction  is  confirmed, 

3.4.5  Progressiva  deepening 

The  second  part  of  tha  third  prediction  stated  that  design  experts 
would  use  the  strategy  of  progressive  deepening,  in  response  to  the 
limited  capacity  of  thalr  working  memory.  The  results  of  progressive 
deepening  show  up  in  the  gradually  elaborated  problem  conception 
schema . 

Progressive  deepening  is  operationalized  as  follows: 

1)  the  problem  solver  changes  the  contents  of  a  particular  slot  in  a 
paradigm  (this  change  excludes  mere  repetition  of  the  contents); 

2)  the  slot  or  its  contents  have  been  mentioned  before,  but  not  in  the 
immediately  preceding  protocol  statement  (this  requirement  excludes 
justifying  statements) . 

As  defined  in  the  task  analysis,  the  slots  in  the  problem  conception 
schema  include  the  Independent  and  dependent  variable,  control  vari¬ 
ables  ,  subjects,  and  possibly  the  setting,  the  outcome,  and  the  in¬ 
structions  to  subjects.  The  following  paragraphs  illustrate  tha  strat¬ 
egy  of  progressive  deepening  in  the  design  experts'  protocols. 

Progressive  deepening  was  observed  in  the  protocols  of  all  design 
experts.  For  example,  paradigm  B  was  mentioned  four  times  in  all  by 
Design  Expert  2,  if  we  ignore  the  variants  for  the  moment: 

1)  The  first  time,  B  was  referred  to  simply  by  its  generic  name  "panel 
research" 

2)  The  second  time,  the  panel  research  was  described  more  fully  by 
including: 

-  the  Independent  variable  (three  colas) 

•  the  Instructions  ("we  would  like  to  know  what  you  Ilka  or 
dislike  about  Coca  Cola") 
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•  «  control  variable  (blind,  i.a.,  no  brand  names  vialbla) 

the  dependent  variable  (description  of  taste;  identification  of 
Coca  Cola) . 

At  this  stage,  the  number  of  subjects  and  the  statistical  design 
are  mentioned  by  the  subject,  but  are  left  open  ("I  would  not  know 
that  right  now").  Five  elements  are  mentioned  in  total  the  second 
time  the  panel  research  was  described. 

3)  The  third  time  was  an  extension  of  what  was  stated  above,  and  im¬ 
mediately  follows  it,  after  the  subject  has  briefly  checked  the 
problem  statement  again.  The  panel  research  is  now  referred  to  as 
"free",  meaning  "with  open  questions".  Design  Expert  2  explicitly 
focuses  his  attention  on  this  less  structured  experiment  first 
("Perhaps  it  would  be  good  to  try  to  do  it  in  two  stages  end  first 
allow  some  open  questions").  New  categories  are  added  and  more 
items  are  mentioned  with  the  old  ones.  The  new  categories  added 
are: 

number  of  subjects  (25  to  50) 

•  treatment  ("have  them  taste  a  bit  and  allow  them  to  go  back  and 
forth  between  colas") 

-  statistical  analysis  ("result  is  a  number  of  dimensions  that 
are  not  too  clear  because  of  a  lot  of  noise;  probably  one 
clear,  but  uninteresting  dimension"). 

Items  added  to  old  categories: 

-  independent  variable  (three  glasses  should  be  coded:  a,b,c) 
controls  (balance  order) 

•  dependent  variable  (describe  differences  in  taste). 

Eighteen  elements  are  mentioned  in  total  the  third  time  the  panel 
research  was  described.  Thus,  compsrlng  the  second  with  the  third 
time  the  panel  research  was  mentioned,  more  than  three  times  as 
many  elements  are  mentioned  the  third  time. 

4)  The  fourth  time,  paradigm  B  was  referred  to  as  the  "more  struc¬ 
tured"  approach.  Since  this  approach  uses  a  different  dependent 
variable  than  the  less  structursd  approach,  both  approaches  cannot 
be  considered  extensions  of  one  another. 

In  this  protocol,  five  slots  «re  successively  refined,  two  of  vhich 
(the  number  of  subjects  and  the  statistical  design)  are  left  open  at 
first,  but  are  explicitly  mentioned.  The  other  three  slots  (indepen¬ 
dent,  dependent,  end  control  variables)  are  elaborated  at  several 
places  in  the  protocol. 

The  protocols  of  all  subjects  were  analysed  in  this  way,  and  the 
results  are  shown  in  Table  IX. 
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Table  IX  Average  number  of  aucceaalvely  refined  aloca 
for  the  four  group*  of  aubjeota. 


Beginners 

1.5 

Intermediates 

3.7 

Design  experts 

3.3 

Domain  axperts 

1.5 

A  Chl-aquare  teat  on  the  total  nuaber  of  alota  jhowed  a  algnlficant 
dlfferenea  between  the  four  group*  for  the  number  of  aucceaalvely 
refined  alota  (Chi-square(3)-13 . 33,  p-0.004).  The  reaulta  clearly  ahow 
that  both  the  lntermedlatea  and  the  dealgn  expert*  aucceaalvely  re¬ 
fined  the  elota  In  their  problem  conception  achemata.  In  contraat,  the 
beginner*  and  the  domain  exparta  did  not  return  to  slot*  already 
filled  in.  Therefore,  the  second  part  of  the  third  prediction  la  alao 
confirmed. 


4  GENERAL  DISCUSSION 

Generally,  the  data  fit  our  hypotheaaa  well.  Flrat,  the  main  reaulta 
will  be  aummarlzed.  Next,  the  reaulta  will  be  Interpreted  In  term*  of 
the  theoretical  framework  developed  above.  Finally,  theoretical  and 
practical  implicatlona  of  the  reaearch  reported  here  will  be  de- 
icribed. 

The  main  reaulta  of  the  proaent  atudy  were: 

•  dealgn  expert*'  goal  atructurea  ware  much  more  structured  than 

those  of  the  beginners;  the  goal  structures  could  not  be  distin¬ 
guished  from  those  #f  the  domain  experts; 

-  design  experts  and  Intermediates  frequently  used  the  strategy  of 

mental  simulation;  the  strategy  of  hypothetical  reasoning  was  used 
leas  frequently  and  only  by  the  design  experts; 

only  the  dealgn  and  domain  experts'  problem  conception  schemata 
contained  general  elements,  supplied  missing  data,  helped  to  focus 

attention  on  the  important  problem  features,  and  abstracted  from 

and  changed  irrelevant  details; 

-  the  problem  conception  achema  of  the  design  experts  and  tbs  inter¬ 
mediates  was  gradually  elaborated  by  a  strategy  of  progressive 
deepening. 
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What  do  these  results  suggest  oonoamlng  tha  main  quaatlon  in  thia 
atudy,  vis,  how  do  oxparta  solva  noval  problems  within  thair  domain  of 
expertise?  Tha  raaulta  claarly  ahowad  that  tha  daaign  axparta  in  thia 
atudy  did  not  bahava  Lika  novieaa.  Inataad,  thair  problem  aolving 
could  ba  daaoribad  vary  vail  by  tha  a  ana  modal  that  described  tha 
domain  axparta'  problem  solving.  Therefore,  whan  knowledge  is  lacking, 
tha  order  in  which  goals  are  accomplished  can  remain  tha  same,  pro* 
vidad  that  not  too  much  knowledge  is  lacking,  as  was  tha  case  with  tha 
beginners  in  this  study.  Whan  too  much  knowledge  is  lacking,  tha 
problem  solver  mainly  wanders  from  one  impasse  to  another,  displaying 
aeemingly  random  search  behavior. 

The  daca  strongly  suggest  that  tha  availability  of  a  problem  concep- 
tion  schema  in  the  form  of  a  paradigm  greatly  helps  to  structure 
problem  solving.  When  an  experimental  psychologist  is  confronted  with 
a  noval  problem,  this  problem  is  first  categorized  as  belonging  to  a 
certain  abstract  category,  e.g.,  "a  multi-dimensional  scaling  prob¬ 
lem".  This  abstract  category  evokes  a  paradigm  that  subsequently 
guides  problem  solving,  i. a. ,  it  helps  interpret  the  problem  statement 
and  specifies  the  general  categories  (Independent  variable,  etc.)  for 
which  information  should  be  obtained.  In  this  sense,  experts  still 
exhibit  the  schema-driven  problem  solving  that  characterizes  their 
routine  problem  aolving  (e.g.,  Van  Lahn,  1989),  even  when  confronted 
with  novel  problems 

However,  the  experimental  psychologist  may  encounter  impasses  along 
the  way  whan  trying  to  design  an  experiment  in  a  novel  domain.  Several 
paradigms  may  ba  evoked  and  it  may  not  ba  clear  which  one  to  choose. 
In  this  case,  the  researcher  resorts  to  the  strategy  of  hypothetical 
reasoning,  imagining  what  the  outcome  of  a  paradigm  would  be  and 
checking  this  outcome  against  the  problem  requirements.  Whan  a  par¬ 
ticular  paradigm  is  chosen.  It  may  not  ba  clear  how  to  fill  in  the 
details  of  that  paradigm.  In  that  case,  the  researcher  uses  the  strat¬ 
egy  of  mental  simulation,  imagining  how  the  experiment  would  look  like 
when  it  would  actually  ba  carried  out.  When  using  the  strategy  of 
mental  simulation,  the  researcher  is  frequently  reminded  of  general 
design  principles  thst  apply  in  this  particular  case. 

The  results  fit  into  our  theoretical  framework  as  follows.  The  design 
experts  have  developed  a  task  dependent  goal  structure,  that  specifies 
what  steps  they  have  to  take  when  an  experiment  has  to  be  designed. 
Their  general  knowledge  of  paradigms  end  design  principles  is  Indexed 
with  respect  to  this  goal  structure,  i.e.,  the  relevant  knowledge  can 
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•tally  ba  retrieved  whenever  a  particular  goal  haa  to  b«  satisfied, 
Tha  goal  atruotura  ramalna  tha  aama  from  one  problam  to  anothar  within 
tha  same  domain.  Tharafora,  thia  goal  atructura  can  ba  usad  in  aolving 
noval  problana.  What  ia  lacking  in  thoaa  caaaa,  ia  the  domain  knowl¬ 
edge  nacaaaary  to  aolva  noval  problama.  Thia  lack  of  domain  knowladga 
axhlblta  itaalf  in  tha  protocola  aa  aaarch  behavior.  Exparta  cona train 
thalr  aaarch  by  ualng  domain- dapandant  atratagie  knowladga.  Mote  that 
"domain-dependent"  in  thia  caaa  rafara  to  tha  domain  of  axparimantal 
daalgn  and  not  tha  domain  of,  for  lnatanca,  aanaory  paychology.  Strat- 
•glea  such  aa  hypothetical  raaaoning  and  mental  aimulation  are  more 
generally  applicable  than  in  tha  domain  of  aanaory  paychology  alona. 
Therefore,  they  can  ba  considered  as  the  most  general  strategies 
within  tha  domain  of  designing  experiments. 

Tha  limited  capacity  of  working  memory  imposes  severe  limits  on  how 
many  goals  can  be  accomplished  at  once,  and  how  many  subgoals  can  be 
kept  active.  When  aolving  noval  problems,  the  domain  knowledge  often 
has  to  ba  assembled  in  various  problam  spaces  by  a  lengthy  aaarch 
process.  Tha  design  experts  in  this  study  often  chose  not  to  go  into 
too  much  detailed  search.  Instead,  they  preferred  to  keep  a  global 
picture  of  tha  complete  paradigm  active  in  working  memory.  That  la, 
they  went  over  tha  same  paradigm  again  and  again,  leaving  details  open 
at  first,  but  gradually  adding  more  detail.  For  instance,  they  started 
by  referring  to  a  paradigm  by  its  name;  when  they  returned  to  it,  they 
tried  to  find  a  general  value  for  tha  Independent  and  dependent  vari¬ 
able;  these  general  values  ware  subsequently  more  specified  and  other 
elements  /ere  considered  as  well  (e.g. ,  control  variables,  number  of 
subjects,  statistical  analysis).  In  short,  the  design  experts  used  the 
strategy  of  progressive  deepening. 

Neither  beginners  nor  domain  experts  used  these  strategies  for  con¬ 
straining  their  search.  Beginners  only  used  less  successful  strat¬ 
egies,  which  did  not  result  in  the  retrieval  of  the  knowledge  required 
for  solving  tha  problem.  Domain  exports  did  not  have  to  use  any  strat¬ 
egies,  since  they  did  not  encounter  any  Impasses.  Interestingly,  the 
intermediates  frequently  used  the  strategy  of  mental  simulation,  in 
conjunction  with  design  principles.  They  also  progressively  deepened 
their  paradigm.  These  results  weakly  suggest  that  knowledge  of  suffi¬ 
ciently  detailed  and  Integrated  paradigms  that  the  intermediates 
already  possessed  enabled  them  to  accomplish  their  goals  in  an  "ex¬ 
pert-like”  way.  The  intermediates'  paradigms  were  not  yet  as  abstract 
aa  the  design  experts',  which  sometimes  resulted  in  the  selection  of 
an  Incorrect  paradigm.  In  short,  the  intermediates'  form  of  reasoning 
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was  similar  to  that  of  tha  exparts',  but  thair  eon cant  of  raasonlng 
dif farad. 

Thase  results  have  the  following  Implications  for  those  theories  of 
cognitive  skill  acquisition  (e.g.,  Anderson,  1987)  that  place  a  great 
deal  of  emphasis  on  highly  domain- specific  knowledge  in  experts' 
cognitive  skills.  First,  this  study  has  shown  that  experts  have  a 
flexibility  that  goaa  beyond  mere  domain- specific  knowledge.  When  this 
knowledge  is  lacking,  experts  can  still  outperform  novices  by  making 
use  of  more  abstract  knowledge,  and  strategies.  Current  theories  of 
cognitive  skill  acquisition  have  mainly  focused  on  the  distinction 
between  so-called  "weak"  and  "strong"  methods.  The  present  study  has 
shown  that  there  may  exist  methods  of  Intermediate  generality,  such  as 
hypothetical  reasoning  and  mental  simulation.  Exactly  how  general 
these  methods  are  Is  a  matter  for  further  research.  Second,  the  pres¬ 
ent  study  has  indicated  how  these  strategies  mske  use  of  the  abstract 
knowledge.  For  instance,  abstract  knowledge  of  paradigms  contains 
information  about  the  general  outcome  of  the  paradigm.  This  Informa¬ 
tion  can  be  used  by  the  strategy  of  hypothetical  raasonlng.  Also, 
experts  have  represented  the  abstract  categories  in  their  paradigms  in 
a  temporal  order.  The  strategy  of  mental  simulation  makes  use  of  this 
temporal  ordering.  Third,  the  results  of  the  present  study  suggest 
that  a  reorganization  of  declarative  knowledge  in  terms  of  organized 
schemata  occurs  after  only  limited  experience  with  problem  solving. 
The  transition  from  beginner  to  Intermediate  In  this  study  may  be 
viewed  as  an  Instance  of  this  reorganization.  Since  the  intermediates 
were  more  like  the  design  experts  than  like  the  beginners  on  most 
relevant  measures,  this  result  suggests  that  the  Initial  reorganiz¬ 
ation  of  knowledge  plays  a  much  larger  role  than  has  been  assumed 
until  now. 

Practically,  these  results  may  have  interesting  educational  Implica¬ 
tions.  Strategic  design  knowledge,  in  the  form  of  a  goal  structure,  Is 
not  taught  explicitly  In  courses  on  expsrimental  design,  which  may  be 
part  of  the  reason  why  the  beginners  In  this  study  had  so  much  trouble 
coming  up  with  a  good  design.  This  strategic  knowledge  is  now  derived 
from  practice  in  designing  experiments.  It  may  be  Interesting  to  try 
to  convey  this  strategic  knowledge  for  the  tesk  of  designing  experi¬ 
ments  to  students.  Presenting  students  with  high-level  goal  structures 
could  reduce  their  working  memory  load.  Studies  such  as  those  by 
Schoenfeld  (1779)  in  the  domain  of  mathematics  have  indicated  that, 
under  appropriate  conditions,  strategic  knowledge  can  be  taught  suc¬ 
cessfully.  Besides  strategic  knowledge,  one  could  try  to  convey  to 
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students  the  existence  of  broad  classes  of  research  questions  and 
broad  classes  of  answers  in  terms  of  paradigms.  A  training  study  in 
which  the  acquisition  of  knowledge  about  paradigms  and  strategic 
knowledga  would  be  separately  manipulated,  could  perhaps  answer  the 
question  why  the  beginners  in  this  study  did  not  use  any  of  the  strat¬ 
egies  the  design  experts  used. 

A  second  practical  application  lies  in  the  area  of  expert  systems. 
Most  expert  systems  nowadays  are  competent  in  very  narrow  domains. 
This  limited  competence  makes  them  very  sensitive  to  slight  changes  in 
input  data.  At  the  same  time,  these  systems  cannot  transfer  their 
knowledge  to  other  domains  and  lack  explanatory  power.  Recently,  some 
attempts  have  been  made  to  develop  systems  that  are  more  flexible  and 
are  better  able  to  explain  their  reasoning  (Larkin,  Reif,  Carbonell  & 
Gugliotta,  1985;  Clancey,  1988).  These  systems  also  Incorporate  the 
idea  of  strategic  knowledge,  represented  separately  from  the  domain 
knowledge.  The  goal  structure  and  concomitant  strategies  that  this 
paper  described  for  the  task  of  designing  experiments  may  also  gener¬ 
alize  to  other  tasks  involving  design.  For  instance,  the  strategies  of 
mental  simulation,  hypothetical  reasoning,  and  progressive  deepening 
have  been  described  in  domains  such  as  software  design  (Adelson  & 
Soloway,  1985;  Kant  &  Newell,  1984),  architecture  (Coyne,  Rosenman, 
Radford,  Balachandran  &  Caro,  1989;  Goel  &  Plrolll,  1989),  and  engin¬ 
eering  (Goel  &  Pirolli,  1989).  It  may  well  be  that  these  different 
domains  share  a  number  of  "design  strategies"  that  could  be  incorpor¬ 
ated  in  a  flexible  knowledge -based  design  system,  similar  to  the 
diagnostic  strategies  developed  by  Clancey  (1988). 
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Appendix  A  Coding  scheme 


Orlantata  on  task  (0) 

01:  cask  requirements :  "so  Che  cask  is  to  say  what  you  have  to  do 

while  thinking  aloud":  "and  you  want  ae  to  do  this  under  time 
pressure?" 

02:  problem:  "it's  a  well-known  problem  at  any  rate:  It  has  received 

quite  some  attention  in  the  press" 

03:  question  to  experimenter:  "I  am  allowed  to  write  down  things, 

Just  for  myself?" 

Understand  problem  (U) 

Ul:  generate  problem  constraint:  "manufacturer  of  Coca  Cola  wants  to 

improve  his  product";  "so  what  people  taste  exactly,  that  is  his 
question" 

U2:  evaluate  problem  constraint:  "but  that  does  not  mean  that  you 

have  to  cake  that  eerloualy  as  investigator";  "may  well  be  chat 
the  Pepsi  Cola  is  preferred  at  a  certain  moment” 

Select  paradigm  or  analogy  (SP) 

SP1:  generate  paradigm  or  analogy:  "then  1  will  go  on  to  a  more 

'difficult*  experiment  on  what  people  exactly  taste";  "I  under¬ 
stand  that  a  panel  experiment  is  a  kind  of  constraint";  "you 
could  do  the  well-known  Pepsl-challenge" ;  "you  could  think  of 
some  kind  of  questionnaire" 

SP2:  evaluate  paradigm  or  analogy:  "maybe  we  should  abandon  that 

plan";  "and  then  palrwlae  comparisons  may  be  useful *;  "if  you 
want  to  Investigate  with  young  children,  then  questionnaires 
don't  get  you  very  far" 

SF3:  justify  paradigm  or  analogy:  "because  if  they  can't  do  that, 
then  there  is  not  much  use  continuing";  "a  panel  experiment  is 
what  comes  to  mind  automatically,  because  if  you  were  going  to 
interview  people  on  what  they  taste  when  they  drink  Coca  Cola, 
then  of  course  you  will  have  nuisance  factors  such  as  image  and 

so  on";  "I  think  I  would  do  it  with  a  card  system,  just  because 

taste  is  so  difficult  to  scale" 

Select  design  principles  (CP) 

DPI:  generate  principle:  "they  have  to  be  able  to  switch,  1  think"; 
"and  finally  they  also  have  to  say  which  one  is  cola";  "I  think 
you  need  quite  a  few  subjects";  "and  then  I  don't  know  enough 
about  details  whethar  you  have  to  eat  a  little  piece  of  bread 
after  that" 
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D?2:  evaluate  principle:  "perhaps  it  would  ba  a  good  Idea  to  try  to 
do  It  In  two  stages";  "and  than  it  becomes  leas  interesting 
whether  they  have  to  identify  cola  or  not" 

DP3:  justify  principle:  "I  want  an  overall  judgment,  because  other¬ 
wise  I  cannot  deduce  which  one  you  prefer" 

DP4:  leave  details  of  principle  open:  "balance  order  and  those  kinds 
of  technical  details,  I  don't  know  whether  I  have  to  go  that 
far";  "well,  that  requires  some  further  consideration";  "that 
panel,  yes  how  large  that  would  need  to  be,  and  the  statistical 
design,  I  would  not  know  that  right  now" 

Pursue  paradigm  (PP) 

PP1:  generate  solution:  I  think  I  would  cake  a  glass  with  a  color 
which  just  doesn't  mske  you  see  any  differences  in  color  between 
the  drinks";  "and  so  you  give  a  questionnaire  and  you  lot  them 
score";  "and  then,  secondly,  subject  gets  a  drink" 

PP2:  evaluate  solution:  "just  to  make  it  kind  of  fun  for  the  sub¬ 
jects";  "well,  with  those  data  you  would  bo  oblo  to  do  some¬ 
thing";  "perhaps  it  is  even  better  if  you  end  up  there" 

PP3:  recall  solution:  "1  have  already  said,  non-relevant  factors  are 
those  image  things,  matters  of  order  of  presentation" 

Evaluate  task  (E) 

El;  evaluate  task:  "is  this  enough,  or  do  1  have  to  go  on,  have  I 
forgotten  something  laporton t?";  "what  other  questions  are 

there?" 

Monitoring  statements:  "I  just  go  on  for  a  moment";  "X  am  thinking 
about  numerous  things";  "I  presume  I  do  not  have  to  explain  that 
fully";  "let's  see,  can  I  think  of  anything  more  about  that 
target  population" . 

In  the  synthetic  protocol,  certain  words  are  underlined.  When  coding 
the  protocol,  these  words  may  be  used  as  a  guide  for  classifying 
statements. 

For  example,  evidence  for  a  subject  working  on  the  problem  formulation 
is  apparent  from  words  such  as  ’question'  and  'goal'  and  from  literal 
phrases  from  the  problem  formulation. 

Use  of  paradigm  is  indicated  by  words  such  as  'experiment',  'plan', 
'method',  and  'lnvastlgatlon' . 
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Verba  aueh  *•  'have  to',  'can',  and  'want'  vary  often  indicate  uae  of 
a  design  principle,  e.g.,  "You  have  to  provide  aome  open  dimenaiona" ; 
"You  have  to  give  an  inetruction  like..."!  "The  flrat  tine  1  want  to 
aeaaura, . 

Verba  in  the  preaent  tenae,  auch  aa  'let',  'give',  'get',  'do',  and 
'have'  often  give  en  indication  of  the  current  atate  of  the  paradigm, 
e.g.,  "three  of  thoae  glaaaea,  let  them  taate,  end  then  let  them 
name";  "You  juat  have  three  beakara:  a,b,c”;  "One  gate  a  drink,  and 
you  eey. . .". 

Adjectives  auch  aa  'important',  'good',  and  'interesting*  lndloate 
evaluative  statements,  e.g.,  "but  that  is  probably  an  uninteresting 
dimension";  "that's  probably  not  so  bad". 
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Appendix  B  Interpretation  of  ptMeeel  ef  Design  Expert  J, 


1  Under ■ tend  problem 

1.1  Read 

1.2  Recapitulate 

1.3  Write 

1.4  Summarize 

1.5  Criticize 

1.6  Generate  alternative  reaaarch  queatlon 

1.6.1  Select  paradigm  Al 

1 . 7  Read 

1.8  Recapitulate 

1.9  Write 


2 

2.1 

2.1.1 

2.1.2 

2.2 

2.2.1 

2.2.2 

2.3 

2.3.1 

2.3.2 

2. 3. 2.1 

2. 3. 2. 2 

2. 3. 2. 2.1 

2. 3. 2. 2. 2 

2. 3.2.3 


Select  paradigm 

Generate  paradlgma  B1  and  B2 

impaaae:  inaufficient  knowledge  to  chooae  between  paradlgma 

repair:  Read  notes 

Generate  paradigms  B1  and  B2 

impaaae:  insufficient  knowledge  to  choose  between  paradigms 
repair:  Generalize  givens  in  problem  statement 
Generate  paradigm  Al 

impaaae:  inaufficient  knowledge  to  chooae  for  paradigm  Al 
repair:  evaluate  paradigms  in  evaluation  problem  space 
evaluate  paradigm  Al  by  hypothetical  reasoning  (evaluation 
negative) 

evaluate  next  paradigm  on  list  (B2)  by  pursuing  the  para¬ 
digm 

impaaae:  inaufficient  knowledge  to  pursue  paradigm  B2 
repair:  evaluate  by  hypothetical  reasoning  (evaluation 
negative) 

evaluate  next  paradigm  on  list  (A2)  by  hypothetical  reason¬ 
ing  (evaluation  positive) 


i?ji: 


3 

3.1 

3.1.1 

3.1.2 

3.2 
3.2.1 


3.2.2 

3.3 


Pursue  paradigm 

Generate  paradigm  A2:  retrieve  from  LTM 
impasse;  insufficient  knowledge 

repair:  select  design  principle  (random  presentation  of 
stimuli);  atop  when  too  much  detail  is  retrieved 
Generate  paradigm  B2:  retrieve  from  LTM 

Impasse:  Insufficient  knowledge  to  choose  among  alterna¬ 
tives 

repair:  avoid  too  much  detail:  leave  decision  open 
Generate  paradigm  B2:  retrieve  from  LTM 
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